"Hi @LostRuins , I've opened this draft PR as a functional proof-of-concept for the Smart Cache feature. As we discussed, I'd really appreciate your feedback and any help you can offer to refine it before I mark it as ready for a formal review. Thank you!"

@LostRuins this PR is ready to be reviewed
Multi chat, image generation and AIHorder have been tested on my Linux + CUDA hardware only

While implementing the Smart Cache feature, I noticed the gpttype_adapter.cpp header
includes these design considerations:

//No dynamic memory allocation! Setup structs with FIXED (known) shapes and sizes
//Python will ALWAYS provide the memory, we just write to it.

My implementation has a few deviations from these strict rules:

Current Choices:

1. savestates management: Using new std::vector<savestate_data>(limit) for
   dynamic slot allocation (3-256 configurable)
   - Alternative: Static array → wastes ~100MB if using only 3 slots
2. smart_cache_lcs_precomputed: Uses std::vector with capacity pre-allocation
   - Alternative: Static buffer → wastes 131KB permanently
3. smart_cache_compute_purge_diff(): Creates temporary vectors for LCS computation
   - Alternative: Python passes work buffers → more complex API
4. get_current_context_tokens(): Returns pointer to C++ memory (zero-copy)
   - Alternative: Python allocates + memcpy → slower, requires 2 calls

These choices favor:

• Performance (zero-copy, minimal allocations after warm-up)
• Flexibility (configurable 3-256 slots vs fixed 256)
• Memory efficiency (only allocate what's needed)
• RAII safety (automatic cleanup, no leaks)

vs strict compliance which would require:

• ~230MB wasted static allocations
• 5-10% performance loss (extra copies)
• More complex Python API

Question: Are these pragmatic deviations acceptable, or would you prefer
strict compliance with the original design constraints?

Thanks @wbruna! All three points addressed:

1. Declarations moved to model_adapter.h - type safety now enforced by compiler
2. vector* → vector - simpler, RAII-safe, removed ensure_savestates_initialized()
3. Removed max_savestate_slots - using savestates.size() directly (single source of truth)

Changes pushed. Ready for re-review!

If in the future this ends up being a half useful feature rather than fully useful due to the potential of needing a LOT more sysRAM, I suppose one option is to specify a maximum context size the user may send over before it gets either truncated before generating the Ctx K-V database, or entirely skipped with the option of reporting this back to the user?

My phrasing isn't exactly the most eloquent, but this is what I can muster for linguistics at this time. The assertion of half/full usefulness isn't a dig at you, but rather about what the user would find as a barrier to entry/usage.

In any case, with all the newer storage methods and RAM speeds and all that are around lately, the need to actually copy it back to VRAM might be almost redundant depending on the latency introduced by the process of loading it into vram. System ram is blazing fast, and I can attest I have in many situations preferred fully manually offloading KV cache to sysRAM to load all the layers on the GPU because the non-framentation is much more effective. In my case, it's a bit slow albeit still faster than framentation across a GTX1070 & I7 7700K. But that just stands to prove that there is most definitely room for "play" and nonstandard ways of holding onto that KV cache.

As to how the faster storage I mentioned comes to play in this: Some NVMe storage devices come quite scary close to the total roundtrip time you'd find for RAM access given the possibly easier ability to access NVMe directly over the bus. That's... in essence no different than the concept of directly attaching your storage to your PCIe bus for that gaming use case. It makes use of the exact justifications and reasons I'm bringing up now.

That said, I'm not knowledgeable enough to actually have a firm grasp on what the sort of latency figures related to this all turn out to. I'm merely keeping in mind that there is those extra layers to traverse, in the end. That's especially true for Python, even if cPython is really impressive these days.

I'm following this, I'm curious what we end up with for christmas this year!

Consumer gaming PCs have 32 GB ram, sometimes 64 GB (like i do), if you don't use mmap to load the model, most of it is free, unless you use MoE experts on CPU or stuff like that. it's with huge context that this feature truly shines

I'm testing this feature using 48GB smart cache, with both AIHorde worker (around 9% cache hit rate, 36h), waidrin (https://github.com/p-e-w/waidrin) and SillyTavern. I can really notice the difference.

About the "speed" of moving data RAM <-> VRAM, well.. even with a DDR3 RAM you would still get better speeds then having to preprocess thousands of tokens in case of cache hit.

About Storage hierarchy and NVMe speeds, You're absolutely right—modern NVMe (especially Gen4/5) has dramatically arrowed the gap to DRAM latency. The challenge with KV cache specifically is the frequency of access during generation (every token decode touches it), so even 50-100µs NVMe latency vs ~100ns RAM adds up fast. That said, for hibernated slots that aren't actively generating, NVMe could be brilliant—kind of a tiered cache (VRAM → RAM → NVMe).

The current implementation keeps everything in RAM because we're using llama.cpp's save_state_kv(), which serializes to a memory buffer. Extending this to NVMe would need a custom serialization path that bypasses the buffer, but it's technically doable.

About Direct VRAM offload vs fragmentation, Your GTX 1070 experience is a perfect example—sometimes predictable RAM latency beats fragmented VRAM/split execution. The smart cache sits in a sweet spot for that: it keeps the working set in VRAM while letting you maintain a much larger "recently used" pool in RAM without OOM-ing the GPU.

Hi @Pento95 thanks for your PR. Sorry I didn't get to it earlier, been busy.

I took a look at this, and that's an massive amount of code you're adding 😅 . I do like the idea, but (and no offense), don't really like this implementation, especially the changes to koboldcpp.py. Ideally koboldcpp.py should not be managing cached context contents at all - that's so far always transparently handled by the backend (like how context shifting or fast forward is completely transparent to the python side, which simply sends prompts).

I think the segment about automatic LRU eviction based on RAM and variable slot counts is also not necessary mainly due to the high complexity. We actually already have the functionality we can hook into for loading and saving states - so the whole side slot system is a little redundant, the goal is just to automate it (it's currently triggered manually)

Okay... so let's take a step back. Your core idea is solid.

When receiving a new prompt, calculate similarity to current VRAM context
If similarity ≥ threshold (default 0.8), use ContextFastForward (cache hit)
If similarity < threshold (cache miss):
Save current VRAM KV cache to a free RAM slot
Search RAM slots for best similarity match
Restore best match to VRAM if found (RAM hit)
Otherwise, cold prefill and save to new slot

Goal: Allow automatic save and load KV states for reducing reprocessing in cases where we juggle between a small set of repeated prompts (like AI Horde usage). It doesn't actually have much to do with VRAM, we're just reusing the KV state from a past previous prompt.

Suggested simple approach:

• Assume 3 fixed KV save state slots, we just reuse them if this feature is enabled.
• Add 2 new utility functions on C++ side:
  • CanBeShifted - extract the first part of PurgeMissingTokens into a helper function. Simply returns true if a context shift is possible.
  • ComputePrefixMatch - as you did, simply counts the number of matching prefix tokens between two sequences, returns percentage matched
• When prompt is received, right before ContextFastForward around here

  koboldcpp/gpttype_adapter.cpp

  Line 3770 in 9720aa6

  bool blank_prompt = (addedmemory=="" && kcpp_data->prompt=="");

• If CanBeShifted is true, do nothing. Allow shift as normal.
• If CanBeShifted is false, calculate prefix similarity with current_context_tokens of current context
  • If similarity ≥ 80%, do nothing. Allow fast forward as normal.
  • Otherwise, for each of the currently used kv state slots, calculate ComputePrefixMatch and CanBeShifted
    • If similarity to any of them ≥ 80% or CanBeShifted, save current slot and switch to that slot.
  • Whenever loading or saving current slot, simply tag the slot with a timestamp. When running out of slots after all 3 are used, delete the oldest timestamped slot.
  • Slot loading and saving completely reuses gpttype_load_state_kv and gpttype_save_state_kv, nothing else is needed.

This should be way simpler and only requires adding 3 new functions, out of which 2 of them are just helper functions for existing code. It does not even need to touch koboldcpp.py except for adding one flag+checkbox that enables/disables smart cache. And it should work as you intended.

Thoughts?

Just my $0.02 on a specific point:

• Assume 3 fixed KV save state slots, we just reuse them if this feature is enabled.
  (...)
  one flag+checkbox that enables/disables smart cache

At least customizing the number of smart cache slots could be allowed without additional complexity: the algorithm would work pretty much the same, and it could still be controlled by a single config (<=0 to disable, >=1 for the number of available slots).